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These scores, then, presumably »are "closer" to'the- actual performance than 
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Individual timing of, inference respdrises showed that subjects' often had long 
pauses during inter-sentence processing, whereas intra-sentence responding 
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• forifiance appeared to -be . key discriminators be^tween s'yc-cess and failure. 
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' who were matched on reading stores with the previous group. These subjects. 

; w^re giv^ six hours of intensive, "individual practice ort the four processes; 
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\ trained people did show improved ^sentence-logic scores (median about A0% 
over the comparison -group);, and i§ a lacge reasoning*problem contained 
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'\which hinged 'upon appreci^ation of verbal subtleties, though, the 
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The investigation supports\he idea of rapidly teaching some "logical 
tricks" in higher-order cognitive operations; but the special training . 
only works >if the'problem material is clean and unambiguous'.' One obvious" 
extG^nsion cf the study is to see if the' same increase in performance can " . 
be produced in a practical-reasoning domain such as troubleshopttng-of 
digital devices.; another extension is to look more closely at the verbal 
subtleties which so 'effectively prevent solution of some large problem'. 
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a standard matrix format was used^nd five rules A^id^ heuristics were 
taught, which were supposed to facilitate , inter-sentence reasoning/ The' 
^trained .people did- show improved sentence-logic scores' (median about 40% . 
over the comparison group); and if a large reasoning problem contained, 
only stra^ig^tforward sentences, then the training was very .effective. 
In fact, all seven^ subjects, solved correctly a 4-dimension, 5-variable 
negative-disjunction problem within' a few minutes. For those problems 
which'hinged lippn apprecia.tion .of verbal subtleties, though, the train'ing^ 
did not help at all ^ - . ^ 

The investigation' supports the idea of rapidly teaching some "logical 
tricks" in higher-order^cognitil^a operations; but the special training . 
qnly works if the problem material 'is clean and unambiguous. Ont obvious 
extension of the study is to see if the same ^increase in performance can 
>>be produced in a .practical -r^easoning domain such as troubleshooting of . 
digital devices; another 'extension is* to look more cToselyat the verbal 
• subtleties which so effectively prevent, solution of soilfie large problem.. 
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j - I. INTRODUCTION 

Here is a logic problem, taken from a publication which specializes 
inword puzzles: » ' " . 

Do you know the ei^ht men who formed the president's cabinet in 1895 
In case you -don't, the •'^Dllowing clues should enable you to -find their 
names (Morton was one' . :^?ir home states, and the. posts' they held, in- 
cluding Secretary of \avy, • , . 

1. Four of the men served the full term, 1893 to 1897, in the 
same posts; the other f5ur— Smith, the Postmaster General, 

■ the Secretary of State, and the Attorney Gerreral— did not. 

■ r • . 

2. ' Lamont was not the Secretary of Agriculture,> nor was he 

from Nebras'ka. . [ / 

3. Herbert was not from Nebraska or New Vr-k.' 

4. -The. Secretary of the Treasury WiS fron. Kentucky. 

/ 5. Harmon was not from West Virginia o*- Massachusetts. 

. " 

' ■ 6. The Georgian, who was the Secretarv zf the Interior, left " 
his post in 1896. . , ' 
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y. The men from VIest Virginia anc Kentucky were. Carlisle and 
the Postmaster General, not -ecessarily respectively. ' ■ 

8.- The man from .Alai)a-a^as not the Secretary of War of 
Agriculture. 

^ . , • • 

"9. Neither Wil's^on nor the man from Massachusetts served from 
« - 1893 to 1897 in the same posts. • * ' 

10. The Attorney General was from Ohio. ' , ' . 

-11. In 1895, OlRey had; only recently assumed the post he held, 
. when Harmon entered the cabinet to "take Olney's-^^igihal post. 

■ ' As you scan the problem, 'tl>e language seems simple' enough; exc-ept for 

a little awkwardness of phrasing 'i-n Sentences 7 and "11 ,; the words are clea 

and the. meaning is definite. But this- problem is difficult. -A typical 

coll ^student. Who' fs- npt familiar with this sort of thing, will not 



fim'-sh it in less than half an hour ('in one night pless. of 22- people, not 
a, single person got it in that time). Graduate students and faculty will 
probably -need fifteen to twefity -minutes; .and without aid, the problem will 
be forever insoluble for a large frac.tion of--American aduUs. Some- of ;the 
difficulties are quite evident: the sheer amount 6f data presented, the 
multiple chains of jinference required for-the answer, the memory load.- 
There is no hope" o*f guessing the right, answer, and common sens^ is'prob- 
. ably insijfficiVit, ' . , 

Yet thevproblem can be solved readily enough, once you know how to 
go about it. And that is an intriguing th-ing afcout problems like this— 
a confusing and 'difficult task may, it seems, be made a good- deal more 
tractable, by certain rules, gimmicks, and heuristics. The solver must 
somehow- structure, the problem into -something that can be worked on in a 
- fairly routine way. If this structuring, and the subsequent operations, , 
can be made less variable and more efficient, then some control can be 
attained over the-solution process. 

In-^ complexity, ^ogic problems^ Tie somewhere between syllogistic^ ■ " 
reasoning itefiis and the reading of plain-text; the major logical relations- 
are inclusion, exclji/ion, aijjd ordering. It is assujne'd that the solver 
can use the relationships in ordinary life to sort oui people and things- 
thus, 1876 occurs befdre J1880; Monday ig earlier in the weel^ than Thursday; 
$10:00 is not exactly divisible by 3; mothers are older than their sops 
but not necessarily older ,than their husband's nephews; a sportswriter ' 
writing a weekly col-umn does not woric for the Daily;^ a physicist will ^ 
remember lj|s high-school algebra;^ and so forth. Linear syllogisms,- or 
three- tp^- series, may be nested inside a larger problen5^or example, ^ 



•■ ■ ' Ted earns less than Larry. \ ^ • - ■ - '• 

■ •• . ■ ^ ' .. ' - "v ^ .. ■ .. ^- *. 

^ „ Who earns the least? ^ 

-Logic problems have certain advantages for research purposes. Th_ 
are fun to do, at lea'st to some people; and -tfrus^they escape the arid! 
o'f 'the sy-no^istic Reasoning item with its ^deadly "sbjne s"^s not P'' 
phraseology. If a solver'^uses a matrix format, as. marty of our subject: 

do, then each response entered 'in the matrix' cait be observfe^ and timed 

' ' ( ■ 

with relative ease, and the subject can see where he is. Often, a's-eqi 

of discrete responses can be' interpreted in terms of the apparent reasc 

ing being done by the 'subject. ' If the logic problem is being solved a- 

a computer terminal, the supporting software can be arranged 'to provide 

a present-status summary, to calculate such quantities as "proportion c 

necessary information already entered," and to suggest which sentences 

ought to be combined to yield new inferences (Bond, Gabrielli, and Rigr 

'1 977 Elegant scoring systems derived from information theory can pre 

duce s^ch indexes as the "discrete entropic cohesion" between problem 

attempts (Watanabe,, 1.969; Gui as u, .1977). ■■■■ f 



•I; -"Logic problem" seems to have no standard meaning in academic psych 
Wey^jjse the term here to rg'p.resent multi-sentence membership proble 
wKich.,ar.^ amenable tp a^;rapir4x forma^. Some puzzle magazines have 
regular, .logic-problem section, and. there are thousands of 'people w 
are regular devotees. Books that/contain various brain- teasers of 
. > have logic problems in . them; but the number of sentences in their 

problems is usually small, and cnlutinn nfton hinnee:- nn a cinnla i 



. '^no^her reason for' s.tudyiiig HumaQ performance logic problems is 
practical: many technix^l: ac-tiWitjes requi>e t4fe of .words 

.'^into Strict symbols. Cdmputfer programming; is the prime example, of"^ , 
course; but there are many others'.* A maintenance technician who attempts 
,to use a. tech manual, mustxonvert the words given there^into discrete, 
actions appropriate to his problem. And here»is a case from the -inven-- 
• tory control domain (.GildeYsleeve, ,1970) . The following, paragraph gives . 
_.the logic in narrative form: . ' , , ^ ' -> 

■ ' . ' • ^> ' 

When the quantity ordered forva particular item equals or " 
, - exceeds the minimum 'discount qliantity and* the order is from 
a wholesaler, give, the customer a discount and make the 
shipment , This^ presumes that there is sufficient quantity 
on hand to fill the order. , * 

If the quantity ordered is less than the discount quantity, 
bill at reglrlar rates and make the shipment even -if the 
customer is a wholesaler. Do the same if the^sale is retail. ^ 

I'^here is not sufficient quantity on hanc^, bill as above , 
sPkip what can be shipped , and backorder the remainder of the 
order . It must be -emphasized that, in this situation also, ' 
_ even if the discount quantity is ordered,.' if the' customer is • 

, ^\ a retailer, the .discount is not given . . ' 

To 'convert this narrative into" a clea:r action policy, the words, must be 

-transformed int!) a discrete-states decision table, as shown below. Then'to 

carry'out the policy, a person has only to observe which of the eight sta'tes 

obtains, and then follow the instructions in that column. .\ . 

. / - . M 2 3 4 5 ■ 6 7 8 

Quantity-Ordered . Discount-Quantity V YNN-YN-Y 

Wholesale • - - • , , Y Y N N Y - N N 

Quantity-Ordered Quantity-On-Hand * . ' Y Y Y Y N N ,N N" 

.Bill At Discount-Rate , * ' ' .X. - - >- X - - 

Ship Quantity-Ordered . •. X ' X X X -"^ - - • - 

Bill At Regular-Rate ' . • -X, XX-. X XX 

Ship Quanti^-On-Hand . --__XXXX 

Backorder Quantity-Ordered Less Quantity-On-Hand ° - - - - XXX X. 

Figure 1. A Discrete-States Decision Table , . 

' for a Word Problem 
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- Many, of -qie processes in' thi^ .words-into-s^anbols pi;pcess are exhifjited ' 
.^Iso in-t'he logic problem... Indeed,- we believe tlj^^^ in an increasingly 
digitized worlds 'the complex conversion from word to discrete syrtibol is- 
\one of the' most important problems jin applied psychology. " • - 
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II. BEHAVIORAL ANALYSIS OF LOGIC PROBLEMS 



/. 



We first became interested- in logic proB-lems becalise of certain com- 

.' - , * ■ ■ " \» ■ . . ♦ 

puter programs which- were designed ta assist'the solver who faced' a well- 

structured verba? puzzle. ^Sihce we had worked for some years w-fth com- • 

puteriz^ trouble-shooting^ids a^nd diagrams, we ''hoped. thalT programs 

■ ' r " ■■■■ ■ ■ • •: ■ ■ ■. . V 

which could handle verbal inputs would lead to more effective fault- 
locating iehavior. A human might, for instance; I'earn. to imitate ;certa in 
features of a word^processing progij^m, and thereby improve performance! 
Findler's Universal Puzzle Solver (Findler, 1973) accepts logical relations 
among variables as jfnputs,\and achieves a solution via recursive.^earch 
through the set of pbssibilities. At present minicomfs'uter speeds, Findler' 
program computes a solution in a second or two. Wang"*s theorem-prover 
program (Wang, 1 961) works by testing "theorems" or hypotheses against the 
input "axioms" or logical relations. The total number of possibS theorems 
is very large. You keep testing' theorems until. you find one that is val id; 
if Proposition P states that the butlfer/?is the murderer, qh the basis of 
logical evidence, -then a theorem stating that the butler is the murderer 
will be valid (Raph-ael, 1975).-, Incidentally, the basic. Wang algorithm 
is so effective that it proved. all_ . the 200 oTr §o theorems of Whitehead 
and Russell 's:' Princ i pi a Mathematica iTia few seconds, on a now-outmoded ,' 
■IBM' 704 computer. Both the Findler and Wang programs require, of course, 
a" human to translate the verbal problem conditions into a format that the 
machine can process. In this respect, Findler's routine is much easier 
to. use, as it accepts string inputs- for the variables, and prints con- 

_ Ok ■ ■ _ . r. 

strained English sentences, as output. 'Findler's program is also, an ele- 
gant pfece of work -from the computer-science standpoint, it has a very 



^^compact search routine, and the whole SNOBOL program is less th^n ^200 

lines long/ ' , . " ... 

\. * ■ ^ . , • ' • • . - • ■ . ■ ' 

•* • , * 

J- Even so\ when we tried the Find! er and Wafig programs with college 
students, subjects were rather^nn^ayed at the input constraints, and much 
preferred a paper-and-penci^ enyironment to the computerized procedure. 
This was partly diie, no doubt, to the si ow. teletype terminal ; ^t took, a 

■ while for the. subject to enter 'a logical relation-^nd if he wanted-a 

' '■ ■ f - ^ ■ ' 

suninary of progress on -the problem, then- there was^oth^r wait for the 

- . . - > . ' ■ • '■ ■ ' 

machine to clank ou-t a response* Often, the student wanted a "micro- 

response" from -the computer; say, whether a given relation. had already ^ 

been entered or not, or a summary of "what I' already .know" about a single 

Variable. Our BASIC version of the .Fin'dler program did not provide for 

✓ ■ - * ■ . ■ 

al] quei7ies of this type. Thus, the machine "aid" was often perceived 
as s'ome thing of an intruder rather than an assistant (Bond, Gabrielli, 
and^Rigney, 1977). . , 

it soon became evident that a few major? actl-yities Were' discernible 
.in. all logic-problem solution attempts, regardless of whether computer- 
ized or. manual environment was employed. After -a preliminary scan of 
tfie problem stem and the sentences to establish the dimensions, a typical 
solution starts by noting logical identifications and exclusions from the 
separate sentences. Each one of these may reduce the ncimber of possibil- 
'ities, and thus will facilitate final -classificatioh of all the variables. 
In the reference problem given dn Page 1 of this report, -every sentence 
contains such. information. Sometimes the logic is .very simple; from 
sentence 2, we can exclude Lamont from either Agriculture or Nebraska; * 
from';sentent:e 4, we can be sure that the Treasury man. and the Kentucky 



maa are the. same. ' But there are. more complex -and implicit, bits of . 
information, too. Sen tence\_l yields 'Several relations; we see there, • 
" that Smjth cannot be the Postmaster Gerveral, thr5>^cretary of State, >or 
: the Attorney General-. &ut we can also infer *^at Smith and these other ; 
three people mate, up a set of fou^- "short- timbrs, " who did not serve^full 
tenns. ' 'This part of the solution 'attempt; is infcra-sentenbe' processing:. 

When all .the separate sentences, have been irNked for their logic, " 
the ppoy em' usually will not be solved. Information from two of more sen- 
tences must be combined, to- get new inferences. In our example^ combining 
Sentences 1 and 6 will allow you to infer immediately that Smith was from 
Georgia^* and was , the- Secretary of Interior. Combining sentences T, 6,, 
S, 10, and 11 yields (among Others) the firm conclusion that the names of 
the four short-timers were Smith, Wilson, Olney, and Harmon. This kind" 
of reasoning .we. call ^ inter-sentence processing; it is often more complex 
than intra-sentence work," because 'the immediate memory load is higher,, 
and because It' is often hard to know just which sentences^should be put 
together. ^ After watching numerous solution attempts, we believed that . 
there was more variability in the inter-sentence processing than in most 
other aspects of the performance. ^ | ' 

. ■ • ■ , . ' / ^ ■ ■ ■ ■ ■ _ . . ■ ■ ■ i 

Some problems have an Ordering feature," and the.' solution depends- on 

how skillfully the ordering- information is extracted, and handled. The 

'„^'. • ' ■ ■ ■' ' ■ , 

"Pie Contest" shown below is rather simple, as our. problems go, but it - 

illustrates the ordering aspect. Our early observations, indicated that 

people differed appreciably on this aspect (3f performance. Some peop^, • 

did r^ot fully utilize the problem datum about the fifth and sixth place 

pies;j tnis hindered.or prevented solution. . We often saw subjects keeping 



, little order' notes or the margin 'of the problem sheet.. All this suggested, 
the -desirability of separately; scoring -the ordering behavior. 
^ - . ■ • . THE •^lE'- CONTEST 



« 



Ann, Bea, Dot, -Eve, and Sue won "the ■fiop' spx prizes in 
last year's, pie-bating. contest at the Center^ijle County ' 
Fair (one of the women wa<s lucky enough to hcf^ two entri.es 
among the six award-winners}. The awards went— not necessarily 
respectively— to an apple pie, a strawberry pie, and ^a ^sweet 
potato pie. From the following- clues, can you determine which 
pie received each prize, and who baked it? 

1; ■ The judges awarded the apple pie first prize and the peach 
pie second; they decided the cherry pie was too sour to 
place in the top five. ^ 

2.- Bea's ranking was lower than A'hn''s bu-Thigher than Dot's. 

3.. Sue did not receive either first or sixth prize, 

4.. Ann has never attempted a fruit pie, 

5. The sweet-potato ^'pie ranked' just below the chocolate pje, 

6, The woman who won fifth place also received sixth priz^. 

On a large problem, the solver will usually need some way of - recording 
his inferences as he goes along. And there will be a "collection" phase 
where the soTver^fias to, go around apd "pick up" all the inferences made 
so far. Wipen this/ happens, the subject will often mutter something like 
"...let's see .just where I am. on this't^ng," Schwartz (1971)- found that 
a matrix format was best; and [Duzzle magazines often incTiide a' matrix 
with "the problem. We therefore decided to provi-de a matrix with each 
problem,, and to encourage subjects to use i.t, even though we k^new that 
5 otne subjects would scribble other kinds of notes as they went along. By 
■this means, we hoped to standa.rdize and control the recording and collec- 
ting variable. . | " ',- 



• Obviously,, these four process^ 'do not exhaust the behavioral domain 

. * .- of the logiifc pufeile. Each "process" can further be $epara*'ted into finer-, 

^"f^ / 9>^airied components; the'level Chosen by the investigator is 'often 'a de- * 
' •"- . ^ ' • -•- ■ ■' • ' ' ' ' . 

' cision of conveniqnce (§ter-nb'erg, etal, 1978).' ' Actual 'solutions 'are/ . 

■ ■ ■' : ^ , ■ - y - 

often multi-lev-^3 affairs. .They have episodes when a rather flickering • - ' 

o ■ 

search oVer the problem , is carried out; perhaps the subject is trynng to" 
* ' . . \- ■■ , ■ . . J \ '■ 

find something that he has overlooked, or forgo'tteri-, or entered wrongly; 

he can look for lines and- columns that are/-nearly filled up;|r-or he may ' 
' ' ■ ' ' -• I ■ , ■ /' • 

"change^ levels" and attempt to' get belx)w_the superficial membership con- ' 



• > 
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r ditians. ^ Nobody, seems to have studied, imagery in logic problem's,- and 
• ^ though our subjects' usually do ncr^ report strong visual imagery, th 

may well visualize spatial lists and arrays wh'en they are attempting to '"'-f 

order things. - V 

■' ■ , . ■ ^ . . > 

•An expert, logic-problem subject' may not need 'a matrix ai<I, and may 

, Y ■ • - 

^ not do. much in the way of recording. Consider again our Cabinet problem 
"on Page 1. -After th'e usual quick scan through it, the expert will oftkq, 
focus^on high-information sentences—those that mention several variables. 
The expert will quickly perQeive the importance of the first sentence,' 
which effectively splits the eight cabinet members into two! groups of^ 

^ four men. Four men finished the JSaS-^g^erm; four didn't/. The short-' 
timers, then* are: ' * ^ ' . " 

^ Smith ' . ' ' ^ . 

Postmaster General' . " ^ ^ - 

. Secretary of State . v ■ . . 

, Attorney General . . - 

What else do we know about the short-timers? 'From Sentence 6, "the Sec- "~ 

retary of the Interior, who was from Georgia, was, also onV of this group 

■ ■ • — ■ " ■ 

. •■■ • ... . -10- ■ ■■ . • ' 

^ - ' ' ,20 . / . . 



* ''-r^J^ of fbur, and ?he can oniy be^Smi th. Also,/frorii Sentence 10 th^ AttorheV 
* general -is frqm; Ohio. Now we frave: , 



'• ./Smith — Interior — Georgia n - . * ' , 

•'^ • Postmaster General \ • . . 

/ ' \./ Secretary of State . . | . ■ 

.Attorney General — Ohfo ' ■ ' v ' 

' ■ ' I ■ ■ - ' ■ ' • ■ > 

~ From ^entelfee^ 9-, the Se'cretary of S^te is from Massachusetts, and cann'ot 

■ " ' ^ ■ \/ ^ / 

be Smith, Wilson, o^:^^armon (S^tence 5); so he is Olney. Then" Harmon- V 

must be, the Attorney General , with Wils.on as Postmaster General. So our 

•• '. 'V • ■ - ■ ■ • ' 

_ ^ shortrtiJTier tableau is already complete: " - :^ 

' . Smith — Interior — Georgia -. ■ 
Wilson — "Postmaster— West Virginia ' .. . 
Olney— Secretary of. State-~Massachusetts ' • 

/ ' ■/ ^ Harmon— Attorney-^General— Ohio 

• ' ' ■ - * " 

-- The remainder of the^problem now breaks easily. (Laipont, Herbert, and \ 

Carlisle are not from Nebraska, so Morton is): ""But this ^pid "expert" 

solution, which actually was performed by^a staff mfmber, utilized a 

rather special focus ^on the first- sentence- chain of reltoning. "ihi's par-" 

■ticu^lar -solver liked to find a solution* without using a matrix or e^abor- 

ate notation, and so. was especially tuned to'cues that encouraged' 1 ong- 

"but- rapid inference chaining. Much practice, and perhaps hundreds of 

problem -attempts, were required to gain . this facility. An experienced 

, .-solver may even recognize the "s.tyle" of the puzzle author; one of our 

staff members believes that he can -identify -problems in Dell Crossword : 

..magazine which are written by Randall WhTpKey. When he sees that Whipkey 

' is the author, he immediately lobks for some obscure little verbal cue' 

^.v in the.problem, and ;fol lows that, intensively, in the hope of locating 'an 

- informative "catch." 



But our main Interest was not directed so much- to spectacular 
solutions -Tike the one just outlined; we .first" Wanted- tp s'ee whether ^' 
separate "scores of intra-sehtence, inter-sentence, and ordering behavior 
'CQuld predict levels of performance on. moderately difficult problems. 



. so, -then further work (^ould proceed to -separating these, three activities' 
into f,inerj:o"mponents. Schwartz (1971) had reported that , a regul||r ' ' 
"logic test".:i«^r'not"sig^^ correlated with performance oh hard 

logic problems; but we hypothesized that^his logic test may have, been 
. too syllO'gis^j^^T^ do, of course, follow the rules of^ • 

'syllog-istic reasoning; but the structuring of the problem, the word-into- 
symbol translations, the arrangement^f inferences, and the assumptions 
about ,the world— these things are not strictly syllogistic processes. 




p" ... • • ~ ^ ■ > ■ III. PROCEDURES < 

A. Predictor Variable^ ' ' - 

Sentence Scores . We_ wanted. three separate indicators of word- 
into-symbol processing: intra-sentence , inter-sentence , and ordering . 
To produce these scores, the^subjects independently derived the logic 
from a set of eleven sentence/sheets. Each sheet had some sentence- . 
material, and a speciallyJ^eled response matrix. All the sentences " 



were taken from problems\yich were not used elsewhere in the study. Si 
of the sheets had only a single sentenpe, and the subject's task was to 
enter into a problem -inatrix all the ^lots (definite "yes") and X's 
(definite "no") that could be inferred from that bne sentence. 'The last 
five of the sheets had two or three sentences on them, and there the 
subject hadfirst to enter the logic from" the separate sentences, and 
then to combine information across sentences for new dot^ and X's. Four 
of the sentence. sheets had ordering implications j so that order- related- 
responses could be separately identified and evaluated. As far as the 
subject was concerned, no special instructions .were given with regard to 
the ordering variable. Two -of the sentence sheets are shown in Appendix 
A; the second one has an ordering feature. - 

Scoring of • the sentence-sheet matrices was done by-counting each 
dot as ope point; each correct X that was separately inferrable was also- 
counted as one point.. - No adduWaJ points were given for row and col- 
umn X's that followed automatically from the entering of a dot. In^the 
ordering score, only those responses were counted on which atjeast /one' 
dimension (e.g., money, age, days of the week) was clearly ordered, and 
where'the-ordering was necessary for problem solution-. 



c ■ 



• The time each subject spent on eac^h^sen;fence sheet-was individually 
recorded by an unobtrusive observer who used a "free-running- second- ^ 
counter. These times are accurafle^jte> within a second or two. Subjects 
were urged to wo^k along •steadfli, but not to rush their work. Once a 
' subject had finished with, a sheet, he could not go back to it. Exactly 
110 inferences 'could be -logically deduced for the whole set of eleven 
sheets; 36 of these Were order-related inferences; and 19 were strictly 

inter-sentence responses. 

•* ' 

. 2. Reading Scores ipx As a reference measure of verbal ability, the 

McGraw-Hill Reading Test^^ given to all subjects. This test J:akes^ about 
^ two hours to complete; it yields six separate scores,' of which two are 
reading^ rates (McGraw-Hill , 1972) . 

3. Discrimination- Scores . Posner showed that closely-timed classi- 
fication tasks could be separated into additive components (Posner & 
Keele, 1967). His , experimental paradigm distinguishes different levels 
.of processing; 'for example, the. letter pair^AA shows physical identity; 
pair Aa has name identity; and the members of pair AB'differ in both 
physical and. name aspects. Typically, the physically identical pair AA 
is classified as "same"" some' .50 or more milliseconds faster than is -the 
Aa pair. Hunt and others have ^plored the possibil ity that this differ- 
ence in "code access" time might indicate the basic information pro- 
cessing ability- of individuals (Hunt et al, 1973). We decided to in-' . 
cljde a Posner-type classification as one of the tasks each subject 
completed. . ' - ' . ■ ' " . 

Our setup used the same letter-discrimination pairs described by 
. ~ . 7 

Posner & Keele (1967); the major techfii-cal,. difference was that our 



. tinifng was voice- keyed instead of fingfir-keyedV After a two second. 

warning, the stimulus pair was presented and the timer was started, 

V ' ■ ■ . . 

The subject looked,at two' stimul us figures in the display, said "same" 

or "different," and the -vocal response keyed an autojnatic digital timer. 

- ^ .y ' ■ ' ' ' 

After a series of calibration and practice trials to smooth out the > 

responses, a counterbalanced set of 48 data-taking trials was given 

for Physically Identical, Name-Identical , and Different pairs. , All 

sessions were individually conducted, in a quiet laboratory room. 

B. Criterion Variable , 

Six prob.lems'" were chosen from materials we had worked with in the 
past; all were taken from published puzzle collections,- though we made 
some minor changes tc reduce ambiguities. Everybody took the problems 
in the same order, with problems of highest, expected difficulty at the 
end. -Every problem was individually timed for 'each subject, with 35- 
minute time limits- on the later problems in the series; pretests had 
shown that half an hour is a practical maximum f^r a p'roblem, with 
undergraduate subjects. ~ . ■ 

C. Subjects 

the subjects were 39 undergraduate "psychology students at a large 
, Western university; each one received subject-pool. , credit, and also was ' 
paid $2.73 per hour for. participating. Five subjects were eliminated:' 
two because they apparently could not solve the simplest logic problem 

correctly, and three because they had incomplete data. ' - . . ■ ] 

" " /■ J 

D. Administration - \ ■ 

\ — ^ . : ' «^ 

• The McGraw-Hill Reading Test was" given- in two group sessions; all 
other materials were adminis^ter^ individua^Hy *■ or in small groups with 



Severs^ observers- in the. room. Since no subjibts had "regularly worked 
logic problems before, about an hour ^nd a halfbf break-in instruction 
was given; during this period, sevpra] simple problems were worked out, 
and the matrix format was illustrated and practiced". Each subject had . 
to finish a real problem while the observers walked around the room and 
obs"erv'ed the dots and X^s being -entered into the' matrix. There were 
individual differences, of course, in how quickly the instructions were- 
appreGiatedf' but 4^ subjects -had to be working efficiently before going 
on to the scored material. / . * 

The sentences and the criterion problems took. most of tbe experq- - 

mental time, with each subjeot requiring several* hours each on these 

' -"\ ■ ■ 

segments of the study-.^ ' On average^ to -complete the eptire series of - 
measurements, a subject spent a tota] of about twelve. hours, and appeared 
for five or six experimental sessions.' Qvfer. a- long .s'ession, breaks were 
given about every 90 minutes^. All subjects appeared to try hard; most • 
seemed to experience some frustration, and occasional elation, during 
the criterion problems. Many remarks were heard to the effect that 
". . . .1 felt lijc,e I was close, to breaking that probTem,.but I just 

couldn't get another big dot."''^ ^' 

. / . • ■ ■ •- _ ^ • • : 



In -computer-aided problem presentation, it is possible to score .a 
problem matrix for "percent information a 1 ready achieved," -and to 
print out this parameter to the solver.- Sometimes the solver already 
has .entered enough data in the>a'trix to. get a. full solution, but 
has not ^'collected" the Relations efficiently. This information- 
collection failure may explain some of the frustration of our sub- • 
jects; perhaps they had already -solved" the problem, but di'dn't " • 
realize it, because.the data had" not been put together across rows 
and columns. . " 



. . IV. RESULTS , - 

The principal data. of the study consist of fifteen scores-from each-- 
subject; these fifteen scores were: 

Criterion Problems Solypd ■ , - - 

Intra-Sentence Score 

Inter-Sentence Score 

Ordering Sentence Score ' 

MdGraw-Hill Reading (6 scores) 

Posner- type Discrimination (5 time scores) 

Means, variances, and intercorrelations of these 15 .scores are shown in 
Table. 1. ' . 

A. Criterion PrbbTems 

The number of cH>:terion problems sol ved. correctly ranged from 0 to 5, 
out; of a maximum ^possible of six. With an average solution score of 
1.5, the problems were obviously quite difficult for this group of people. 
Nobody solved the final problem in the set of six, and only two of the 
^final set of 34 persons got five correct. In general, our predicted 
order -of problem difficulty was accurate,' with .most of the solutions 
being achieved on the'first two. Since the subjects filled out a' matrix 
for each problem, it was possible to calculate part scores on each prob- 
lem, and then to sum these over probl ems. ^^^^ese part scores, however, 
proved to be very highly correlated with a simple count of number solved; 
so we simply us'ed number solved as the criterion variable. 

B. Sentence Scores . - . . • 

..All three sentence, scores had rather high' variances; there are 
marked individual .differences in translating words into symbols, and 
all three were hi-ghly related to the number "of problems solved, with 
r's ranging from .61 to afbgut .75; two of the, scatter fJl^ots are shown 
in Figures- 2 and 3. 
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Variable , . 


Mean • 


S.D. ■ 
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3 , 


4 


5 

> 


6 


7 
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9 


10 • 


.11 , 


12. 


13, 


.' 1^ 


15 


1. Criterion Proljs 


■•I.5: 


1.46 


.69 


.75 


.61 


.31 


.11 


•55. 


.48 


.32 


'.5^ 


-.42 


-.11 


-.20 


-.53 


-.28 


2, Sentences' 


60 


18.85 




.82 


.78 


.30 


.11 


■ .42 


.i6 


.44 •■ 


.56 


-.22 


-.04 


-.10 


-.31 


-.16 
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15 


■ 8.46 






.61 
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.29 
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.23. 


•33. 


'..39 
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1.81 
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11. Letters (Diff) 


729.3; 


101.12 






















.83 


,86 


-.31 


. .1^ 


12. Letters (PI") ' 


6^7.3 


100.82 




















ft 


' * 


.93 


-.27 


.15 


13. Letters (NI) 


.717.1 


114.92 


























-.09 


.'50 


Ik. D-P3< ■ . 


83.3 


57.18 




























.42 


15. WI-PI ■ • ■ ' 


.70. i^ 


41.49' 































- Tatle 1. Means, Standard Deviations & Correlations 
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The highest correlatiorv obtained with number of problems solved 
was. with the l>nea'r ordering sentence score, which had anV. of nearly* 
• 75, The predictability at'the high end of Figure 3 isVemarkable; , 
•of the\eight 'people who solved three or more criterion problems, all 
were above the mean on ordering- And only one subject who had an 
ordering score .of 16 or more failed to get at least 3 criterion prob- 
lems* In fact, the predictability of overall criterion performance 
from sentence -inference .is so high that scores .from a single sente^nce 
were correlated .74 with the criterion problems, . ^ ^' 

Altogether, the data indicate^ that a rather high sentence score . 
is a necessary but insufficient condition for solving the. criterion 
problems,^ We were surprised at the generally low proportion of infer- 
ences that the subjects attaine;d from the single sentences; the median 
subject managed' to get only a little more than 50% of the total number 
of correct dots and X's" that were logically deri'vable; the top score 
achieved was 94 out of 110, or about 85% of those possible. Many sub- ' 
JectsV then, had trouble extracting the logic from the English sentences, 
and so their later. solution efforts on complete problems were doomed to 
failure. There were some very low sentence scores; for instance, five 
people got zero on the interrsentence measure, which means that they ■ 
missed all the inferences -derived: from "two or more', sentences; so all V /]. 
five would be expected to fail any problem which depended on this kind 
of process. The ger^erally high correlations between sentences and prob- 
lems suggested to us that the 'teaching of:"Togic extraction" might be 
the best way to promote excellent performance on logic prpbTeifis^ 



glntercorrfelations. among sentence 1, sentence 2, and so forth were 
generally positive and moderate; f(i^ instance, sentence -6 had, a median^ 
correlation o-f .46 with. the other ten sentences. There was one excep-: 
tion to this: "Sentence A," the seventh in the series and the first " 
^ multH-sentence sheet, was^ correlate,d negligibly with most other measures 
Perhaps the material on this sheet had a technical or' .semantic defect; 
more likely, the^ inter-sentence skills are more difficult and'less prac-. 
ticed, and the subjects experienced some cognitive strain when first 
encountering combinations of sentences. In any case,'these moderate- 
correlations ^between sentence sheets suggest the' existence of a reliable 
dimension for assessing and [Dredicting problem performance - 
C. Reading and Discrimination Scores ' • ~ 

As can be seen from the correlation mati4x* in, Table Ig McGraw-Hill 
Reading scores correlated from .11 to .55 with number of pj^oblems splved; 
■this was a little higher than we expected*;" rn" a college pbputa'ti on 'which 
was presumably selected on reading. Time, scores on the "fosner- type 
classification task also correlated negatively -with problems solved and 
corpel^ed^less with sentence^ s,c ores; for later analyses,. the discrimin- 
atik^time scores were subtracted from a constant, so that a high score " 
w^ a\"good"' score.- ■ In accordance with Poster's results, physically- _ 
identica^letter pairs yielded 'the sho^ medians. 
(OT"msT)..An incidental finding here was that discrimination response 
times for female subjects tended to be-'slightly' lower than .thos<e for 
males. Two difference scores were calculated. Name Identical - Phys- ' 
ically Identical and Different -^Physically Identical . The reli-abili-6y 
of these difference scores should be appreciably lower than the con- 

' • - ' a - . • ■ 

stiuent discrimination -scores • ' ■ : . ^ 



D. Specific Responses in Sentence and Problem Matrices 

With a mean sentence response score of only about "half of the- 

possible inferences, and with -a. difficult criterion set, it was inevit- 

. ^ble that some' inWences would seldom or never be achieV&d. ^This was 

the case -with several of the inter-sentence relations; again, the data 

^show that people have trouble in combining logic across sentences,, and 

in converting- this logic into a matrix record. As we reviewed the answer 

matrices, it often seemed that subjects preferred to stay at a super- \ 

ficial -level when combining sentences, and avoided the "deeper!" and more 

complex relationships. Maybe they did this because a superficial "one- 

pass"- reading, was often sufficient to get some inferences, and so 'the 

subject would be reinforced for such reading. Right now, this is only 

■ ■ -' • ■ .-f 

a' conjecture; but the matter should be subject to experimental investi- " 

gation'of the "depth", variable-. • "Depth of processing" has been shown to 

-affect memorability of learned list material (Craik & LockhatU, 1972); ' 

■ perhajDs: ^' subject ; '^li^ixal^^ deF)thv"-^rv^^T-uency in 'sfi.ifting:;from"X)ne ";. . 

level' to another, can influence solution efficiency. Maybe explicit ' 

training in analyzing' phraies" and- sentences at different '"depths'" would 

'alert subject to the, possibility of T)'u'r'suing more than one leveB^f 



analysis, and would promote .better solutions". .We have enough materiaT . 
to score .proiDl ems for depth-required- to-solve; and we- general ly-'know/^ 
whether a given' matrix'entry is. derivable from superficial or subtle' 
considerations. ! ' ■ . ' . .•• •1. 

Each subject contributed hundreds of timed responses. And there • 
are so many sequences and response rates in the records that. we have not' 
yet thoroughly* analyzed therfi.. Early in the study, we decided not to ' 



pressure the' subjects unduly about their rates, of responding; but 
often there were unmistakable ^slowdowns when inter-sentence processing 
began. " / ' ' ■ ^ - ^ / • 

E. Factor Pattern 



'One should not take seriously ^factor analysis of more than a 
-do^en scores on 34 subjects; but we did put the correlations from'^ 
Tab! is 1 through a principal-components'-analysis and yarimax rotation, 
using the .California State University packages. The jJoadfngs on the . 
first three factors are shown . in Table 2. ' 

. * . - , ' ■ • . ' ■ , 

, ' " „ ' ' Factor 1 Factor 2 Factor 3 



Variable - 








1. Criterion Problems 


-.63 


-.06 ■- 


-.10 


2. " Serjtences 


-.32 ' 


-.05 


+;io 


3. Ordering , _ ' 


.15 


.22 


+.13 


.. 4. Inter-sentence 


* -.43 


-.13 


-.05' 


. 5. MGH RRl ■ . 


.06 ' ■ 


-.41 


-.70 


6. MGH RR2 " . - 


.18 ■ 


-.17 


^ ,-.■89 


J. MGH - I. ■- ' 


■ .06 . 


-.18 


^ -.28 


8. MGH -■•II 


,. '>^- .00 ■ 




■ -;04 


9. VMGH - Iirv ■ ■• ■^ " ■ 


.'•--.06 " 




■ '-.03- 


10. MGH - Total. 


•.QB 


-.27 


-.07- 


^11. Letters (Diff) 


-.29 


-.80 


'-.33 


^ \T2. Letters (PI) 


'-.11 


-.96'. 


-.05 


. \2. Letters (NI) 


-:10 


-.96 


-.11 


IV D - PI 


-.29 


' .28 


-.64 


15. NI - PI 


, -.01 '-■ ■ 


» -.31 


■ -.42 



* ^ Table .2. ' Varimax Rotation of 

- . V tfi,ree,Princip,al-Component Dimensions 

The factor pattern certainly seems plausible. Problem variates 
(criterion .puzzles and sentence scores) define Factor 1. Factor 2 is 
clearly a Posner-discriminant dimensq"on with three loadings from .80 to 
..96; Factor 3 has two high weights (.70 and .^9) for the McGraw-Hill 



reading scores. Two of the Posner difference scores also load on this 

factor. ' _ • . " ' , 

-. t • . ■ ■ ' .' . 

According to the "discontinuity hypothesis," .there is a rather 

clear demarcation between fast," over] earned, automatized processes,^ such 

as the discrimination skills' i-n the Posner task, and less wQll-practiced 

mental operations' such as our sentence-logic extraction. The factor 

structure obtained above is generally in accord with the discontinuity 

idea; and we have noted before how slow and haltingis the rnter-sentence 

exploration and translation. There are appreciable correlations between 

"good" scores, whetheV the scores represent "fast" or "slow" operations; 

• : , ■ V - • ; V ■ . - .\ ■ . - ; 

but the scores do cluster into rather distinct classes. 

We were surprised at the .28^and .46 loadings of Variables 14- and -15 

on Factor 3. If Factor 3 is primarily a reading^skills -dimension, jthen . 

why shou-ld difference scores from Posner classification load so. highly on 

it? AssufTH-fig that the, .28^ and .46 loadings are not artifacts, we"; con- 

jecture that time difference .between. .Physically-Identical identification : 

and i'dentification of the'other stimuli load's primarily because of rat6 

* ■ ■ - : ^ '. — 

scoring features in t)ie scores involved.. Factor 3 has its only two high 
loadings on McGraw-Hill "Reading Rates", not on McGraw-Hill total, or^ 
McGraw-Hill vocabulary. Perhaps the, scanning features that determined- 
Reading Rate h^ve some basic opertions in common with Posner difference - 
scores. The data from Hunt et al_ (1973) also suggest that some timer 
difference -scores can be.more ^'nformatiye- than ;sVmple reaction time. • \ 
Obviously, a next-step in thi's- area is to confirm the difference-score 
phenomenon on' a larger data set, and toJcontfol experimentaTly the scan-' 
ning "requirements in reading and in classification tasks. The search. 



,Tor Dasic informatiori-pfocessing para'metets - is' tantali-zing and elusive. 

McGraw-Hill 'Reading Rates ?>re determfned 'o'n. the basis of rapid 

' , . ■ ■■ ■ . 

. ' •. -t. . ' 

word recognition; so if a subject were operating under a high-speed ' 

instruction, *it "would not be necessary for the subjecf'to' comprehend 
-the McGraw-Hi/l sentences ^in .their entirety." "JSinstead,. 'the subject could 
search-for a crv^cal key word or two, and give his item response as. .. 
soon as .a key- term was obsejv^d. There mi gh-tN^^en, be natural simi- 
larities to the Pos'ner difference scores, because a "fa'sl: noticer" on 
the. Reading Rate" should be one with: a low difference score from the 
Physical-Identical baseline. . This- interpretation deserves checking 
under differing Reading Rate time pressures;- We should expect that the . 
loadings of the Posner tasks should be reduced as the McGraw-HilT'^task 
shifts from emphafsis on speed to comprehension.. 
F. Practice Effects '-. . ■ ^ 

.Both problems-^and sentences got ha refer as the sessions went along; 
SO iwe caryiot assess accurately the impact of the several hours of prac- ~ 
t i ce . , ■ All ;^^cts , tlrbugh, were quite x.omfarta'ble with -the matrix "format 
after a few sentence sheets;, and they all learned that a nearly^fil led- 

u^ matrix was a Sign that a criterion problem was about to break. A 

• ■ ■ ■ .... ' . ■ "v 

few students came. back and asked to finish problems that they had not 
completed; and some said that ^FTgy had gotten into a problem so much . 
that, later.that day,, they found theraseTves stilT thinking about it, 
and'perhaps seeing a new inference or two while driving home. In our ' 
experience,' this kind of -ititerest in an 'experimental task is rare, and 
extends quite beyond the 2ei garni k effect. > ^ 



V. A TRAINING DEMONSTRATION , 

' "Y 

Once we knew that criterion problems were so highly dependent on* 
sentence-processing components, it seemed worthwhile to try to teach 
these skills. to a fresh set subjects. In a few houris , we- probably 
could not materially improve McGraw-Hill Reading or Posner-type classi- 
fication capabilities, but we thought that the word- in to- symbol activities 
might be subject to some rapid training.^ In our own working and. coding 
of many logic puzzles, we had developed a little set of rules and heur- 
istics which we had fountl to be effective; and tliesrfe techniques seemed 
to be. eminently definaye" and teachable. - 

The demonstration was/-frranned for only seven subjects, on the . ' 
(ground tha€,_jf you 'can't demonstrate your technique on a handful 'of 

i \ ' ■ • ■ • • • - ' ' ' ■ 

people, then therte is little reason. to think you 'can do it with a hundred 
people. We chose seven more undergraduate psychology students whos'e - 
mean McGraw-Hill Reading ■ Scores .were matched to those, of the original 
set of 34 subjei^ts. This group received exactly the same break-in and 

■ ■ ■ -S ' ■ 

criterion-problem procedures as the previous group, but they also went 
-through six hpurs (one day) of intensive training in sentence-processing 
skills. . , ' , . ^ , V . ■ 

During the'training, typical -problem sentences were taken one at 
a' time, and the instructor worked with the trainee until all inferences 

were, correctly extracted. Sometimes it was necessary to give hints and 

— . ■ c 
prompts. All trainees learned to .work very carefully and rather slowly, 

since^they saw that- the criterion was "tct^l correct ^dots and X's j" . 

rather than some speed index. During the last .half of the training. 




\ 



thre^ -complete problems were workedr if the trainee got hun^^ up, then' 
. the instructor gave only, enough of-a. hint to get the process going ' 
again; sometimes it was sufficient to.point to one of the heuristics " . - 
on the board.. * - . . • . 

^ A. The Training Program ^ ^ - 

■ ^ ^ • , Five. ru|es were explicitly taught for aiding the :inter- sentence ' ' 

data-co^ection aspects of solution. These:^^^^^ [■ 
- . . ' ~ - Dot Combination, Suppose'a dot (definite "yes") is entered ' 

• ; • ■ in a problem matrix; ta use our cabinet>oblem again, ^you discover that 

• Smith is from Georgia, and was Secretary of the Interior. It then follows. 
- that "Smith is everything that- Georgia is;", and vice versa, F.or example, 
if Smith and Georgia were on two rows, then every dot and every X in ^ ■ " 

• ■ ■ Smith's rovy now . applies to the Georgia .^^^^^ Th^s rule may appe^ o^W, 

systematical ly, ^perhaps because one of the dotted- vartables would be in ^ 
. a row, and the other in a column, There is a bit of a knack to applying ■ ^ 

; you have to keep track of the ones you have covered. We taught our sub- 
jects to use tick-check marks for this purpose. 
. Choosing Sentences to Comfainp, Suppose you have already' 

entered ajl your intra-sentence logic" into the matrix. Now to find out 
\ ., -whetherX^combine two sentences, say A and B, for, those sentences'- . ^ 
that ha^e; . : . 

. • : r one or more shared variables; , " . - .• , 

(b) the shared variable has to be a--"positive" in A • ' • 
. • , .^nd a "negative" in B; -if this is true, then a 

... _ - ^^"^erence can be'obt^ 

^' o^l^erwise, nothing can be learned from combining. ^ 
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• ThiS' rule is easily generalized to three or more sentence^; in 
fact, a computer program was written which does this automatically for 
a given data 'matrix (students did not use this program as^ an aid; if 
they had, solution would -often have been instantaneous). ' ^ '• 

3. . Hiqh-Informati'on Sentences . Many problems have a long sentence 
or two in which every dimension, er nearly every variable, is incli/ded. 
These should' be especially studied, because they often "split the prob- 
•lem'intp two parts," or otherwise lead to exclusions that produce many 




matrix entries.. One example is Sentence 1 in the problem, on Page 1. 
Here's another illustration: . , 

' The Collins and. the Jones boys-won events before^ ■ .. 

Brenda's son, but after Steve and^the Allen^boy/' ' ^ 

' If there are five boys in this problem, and all five are mentioned 

here, there are many inferences- that flow from this one sentence^- A ,- 

close look at the ordering data indicates that Brenda's.son must 'have 

' been 'last, that neither Steve nor the Allen boy c-an be lower than second, . 

with' Collins and, Jones in third and fourth, places. • - 

4.. Partial but Effective Exclusion . You are woriqng a problem 

witj a five-variable set of people who work for five different newspapers, 

and you know that one of the five, say Jim, is either on 'the Daily or the 

Press; but you vdon't yet know-Which one of these it might be. . Now you . . 

can exclude from Jim any shared exclusion between -the Daily and the • 

Press. If both the D^ily and .the Press, for example, -do not have a-. . ^ • 

soccer-column writer, then Jim does not write the soccer-cqlumn, regard-^ 

less.of jjhether he ends, up. on the Daily or the Press. This logical rule . ^ 

is not- often recognized by the ordinary educated person. In this, respect ^, 
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It resembles Wasbn's modus toll ens inference task (Johnson-Laird & . ' 

Wason, 1970; Wason, 1968), whicK is-extraordinarily difficult -for 

educated adults. ' "• , * . • , . 

5.- Partial Ordering . Trainees are encoufag'ed to set up little 

ordering schemes, where that is possible. As. one example: ' ' • 

Johns's event was before the horizontal bar event'; > " 

there were three .events between thek-- 

] ■ -Now if the solver knows that there are five events and realizes- ' 
J .- ' ■ . ^ ■■ ■ ,T 

fully tjie implications of this sentence, he' will see that John's event . 

was first, and the horizontal bar competition was last, (ginci den tally, 

when this sentence is combined with the one atjout Brenda's son, just 

■ < ■ ■ 

cabove, we see that John must be John Allen, that Steve was the second 

. ■ ' ' ' ' ' L * ' -■ 

. competitor, and that Br"^nda's son was in the horizojTtal-^ bar event).- 

In the course of the training, day, numerous cases arose when these 
five rules could be applied. A list of them was put on the board, and/ . 
the student was- advised to ^ refer to the list,- and perhaps work.thVough 
It, whenever a problem was at a stickinfl ppiiit. - Ouririg tlie.last hour or 
so of training, a moderately difficult problem- was worked fay the trainee, 
. usually without help. - ' * ' ' . ' . ' ' 

After their day of specia-l training, the subjects -took the same 
six criterion problems attempted by the. earlier groijp. They also took 
two additional' problems, which Schwartz (1971) 'had given 'to 38 Ss. at 
W^yne>State University.. One of these extra f)robl eras was "conjunctive- > 
positive,'' with three dimensions and five values; the other was "conjuhc- 
tive-negative," with fpur dimensions and. five values. These problems-- 

; 5.. • Actually, the "negative"' problem as' published by Schwartz is garbled- 
. and insoluble; fierhaps there. was a clerical error in transcribing 

it for the journal. Our revised version contained 13 sentences, and' 
is shown in Appendix B. .> . " •■ . 



were added because they were large but they were atso "clean"— that '"is, 
^\ ■ . • . . . 

there were no verbal subl^eties- in the^ sentences describing the' relations 

Here is a sentence from, the negative-c&njunctive problem: >. 

Neither the Japanese nor the Englishman owns a h:;^ena. 

Jhe matrix entries here are obvious, and will be dpne by nearly every 
trained subject. So even though there are 13 sentences like this," there 
wMl be little^hesltation in entering the dots and X's. \ ■ 

B. ■ Results V ' '. . \ 

The ^trained people entered, correctly 73% of the possible sentence - 
. inferences--a notable -gain over the 50% or so correct in the original .- 
■group. . On the old criterion set, they "came closer" to. filling up 
-matrices apd getting problems, on the average; but their performance 
: tbtjail ^completion scores were identical to those of the previous -group of 
subjects. (a^rage of r.5 solved, oiit "of si-x possible- solutions). • On the"' 
two "large but clean" Schwartz problems,- however, all seven subjects 
solved each problem, >erfeQtly ancf\quickly. ■ 

We interpret the main result as follows. . The Schwartz . problems, 
though large, and unwieldy to. a neophyte, yield readily to the'matrix 
technique and> the five processing tric'KS that we taught. Hence, we got 
rapid and perfect performan'ce on those two problems. Our. original ; cri- 
terion problems, though, often rest -on fair^Ty^scure relations which" 
can be understoocl by the subjects, when they are -po in ted out," but' cannot 
always be perceived readily by the subjects. Tliese more subtle factors ' 
are called implicit relations by Pol ich & Schwartz (1974) ;. those'authorst 
found that errors in formulating tliese'' relations were much more frequent. 
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■ ' *and much more decisive, tifiar>^i^xpliclt' or. surface relations in the 

: problem. It is true that -our train;Tng helped, even in the subtle prph- 
: lens; the trained people correctly made -a larger pro'portion of the 

matrix entries; they made fewer errors; ttiey kept working; the list of 
a five rules always gave th?m something to: do,- and so they made progress. 

But they ^tiTl missed a- key inferencl'-^r* two, enough to prevent total ' 

solution. " • • ^ * . . 

Ordering scores and inter-sentence scores were also notably 
improved^in the trained subjects. There were no zero scores, and 
rather few. erroneous inferences; nearly all mistakes were omissions.' 
On one of the criterion problems, an observer sat. next to each 
' solver and timed every matrix entry. that was made. The resulting record 
then gave a timed solution trace. thro ugh the problem, "and permitted 
response-rate determination at different stages. There are strong 
hints in this material that, for the direct declarative logic from the 
..sentences, the response rates are fairly typical across subjects. 
Figure 4 shows fragments of time records for two 'subjects who worked 
on the pie-contest problem. In both cases,' the early, regular entries 
are due: to the straightforward recording from- the sentences. After thi'.s 
chore has beep accomplished by the solver, there are-often some delays; 
presumably, the time is spent in organizing an inter-sentence search 
strategy. The effectiveness of a special technique or aid can be; eval- 
uated by the control that is achieved over such delays; in the present 
•instance, there was still much uncertainty by the trained subjects after 
..the easy matrix entries were skinrned aff' the top;: "they took a lot of time 
to figure put what to do next. . Y ' 
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Fig.'4 Tinie /'Record for Two Subje^li Pie Problem 
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^ the tw6^-^?fi^rmances in Figure 4 show the typical "early^ 

*spurt" of inferences i. fonowed by later pauses of many seconds. A 

qualitative review of the re'cords, though, suggests that J. L. was not 

only going faster in the early- stages;; -his solution had a better-sense 

of direction and .focus, and his search for /'things to "combine'^ seems to 

be easier to follow, and to mate mor6 sense. -Subject L. L. was making 

inferences, :aTl right; but they -were logically "jerky," and not well 

connected with 6ach other. So., 1. L. could not find the crucial "dots" 

he needed, and did not have a sharp search plan when his separate bits 

' - ■■ ■ .■■ ■ |. • p'- 

of informati OR proved to be insufficient. . . J 

It is easy to. become fascinated vyith- timed protocol data, like this-, 
.and. to see processes that 'may not be general... However, we must mention 
pneothing which struck- us as >^e- went over the records. It is this. , - 
.When ■a subject starts to ejiter dots and XVs- in the matrix, he'pr she ' . 
does this for awhil e, and then si bws down . Our most successful peopl e 
■seem^ to be those who keep going , and keep- entering things. We think ■ 
now that this variable may be part motivational, part habitual , and 
part cognitive, and is the sort of aspect usually, referred to as 
"concentration,." or determination, or mental energy. • Perhaps this 
aspect of performance also would be subject to modification, via train-- 
ing and. imitation of problem-solving "models" who^-enter .data, at steady ' 
rates.- ■ ' .' 



V.I." DISCUSSION 

. .Our res'u-lts Were. positive 1n demonstrating the centp^ality of 

■ ■ ■< ■ ■ . •' , • ■ ■ . . -. ■ a. 

sentence processing in logic problems, and somewhat encouraging with 
ri^spect.. to radically and quickly improving these processing- skills in 
difficult problems. It could.be that our training method,, if continued 
over several days or weeks,, would' have finally produced, much better 
perforniance on difficult, subtle problems.. ..We are inclined to the vi'ew 
that, if the verbal and semantic subtleties are causing many of the: 
" problem hangups, then -analysis should be directed. to the Subtleties them- 
-sel ve^ . 1\ logical n^t project, then, would be, to assemble, the hardest-. 
'*^°~^^'^"'.^y^ Inferences from the present,; data", to frame plausible conjec- 
tures about Tthe reaspns for difficulty, and- to test tfie . coniectures by 
systeinattc 'irariation of the material s. .We already know that f aul ty ; 
syllogistic reasoning,, though it undoubtedly occurs, i^s not- a major- 
source of failure to our subjects. • -i^ 

.Theattempt to teach problem solving had an impressive transfer 
to the two Schwartz problems: perfect performance was attaine^, when, 
our. matrix method and heuristics were.applied to a complicated total 
situation,.wh-ich was made up of many simpl^e sentences. If this result 
can be confirmed on a *l.arge'r- sample of people, it should beWmeaningful ■ . 
to applications in such areas as troubleshooting.!-' Troubleshooting in 
electrogic and mechanical equipment is. often a di'ffrbult and critically 
important task; much time is spent on training. On te;ch manuals, and 
on aids of one kind another. Yet good troubleshoot'ers are as hard 
to find as ever. The aids provided to them are often ineffective; 



and the cris'es; continue. When a" draft copy of this report was bein^ 
reproduced^ four Xerox ..technicians were working^on a recalcitrant 
copying machine; and they ftnany called their o^fice^ for further. he*^ 
In our present framework, the electronic technician who is taught 
"how to troubteshoot" is like -one of our "trainecl" subjects. The tech- 
nician realizes the logic of fault "isolation pretty well ; he knows ,- . 
generally- how to interrupt a signal chain w/th critical tests; but he 
is operating^ on a ■complex "set of sentences".- .in his head, and in his 
manual s» These sentences may contain 1 arge amounts - of information, 
all right, and the technician, extracts some of it; bu^ there are sabtle 
,.ti.es in his "sentences" which he has'not yet .appreciatedi or has for- 
gotten.. As lohg as these exist, no". logical tricks will work; the. ' > 
problem matrix will remain in an.Jncomplete -state; and the trouble will 
■not be located. When effective troubleshooting devices are produced, 
as occasionally happens., they work just because the "sentences" uhder- 
lying them are cl.ean an'd clear and; the fa-uU-isol'ation behavior can pro 
ceed with the. -certainty . thaf't it is converging on the problem . The 
quality of the given technician ';s "sentences" could be assessed fay the 
proportion of correct inferences that could.be 'drawn from his available 
source material. We could then, in principle at-least, e'stiniate the 
Ifkelihood of a given technician ever finding certain kinds of trouble. 
Certainly, the people who' depend on .technicians, and who.provide their 
training and reference manuals, should be interested in such an -estimate 

More research should be done on the highly-practiced and" efficient 
solver of logic problems, since the behaviors exhibited there are really 



what tlf(^ applied psychologist wants. We noted earl ier one "expert" '■ * 
solution to the -Cabinet problem; that performance had a flow and 
elegance, that seems to be lacking in our rather pedestrian matrix 

. • • • • .... • , . • . . ■ . ^ 

technique. Maybe our matrix ski 1 Is, Which we~have showa are qui te 
teachaBle, should be considered as a prerequisite to a" smoother and 
more rapid expert approach. From our staff experience, there is * 
reason to believe that people who become very fas 1^ on the matrix -busi- 
ness can go on to expert- t^e solutions. If this proves to be generally 
sxD, then the matrix technique would' be supportive to the other- cognitive 
operations, and would not be the major technique used by the solver. 
Perhaps real fluency in the matrix skills could be taught in, say, a 
week or less of intensive practice; and^hen the course could/go on' to 
subtle relations. / / ' ■ 

- . In an impressive series of studies, -Robert Sternberg and his. 
■ associates ^ve analyzed syllogis'tic reasoning ijito, a series of about 
ten operations, and have been able t/ estimate the time spent on each 
operation- (Sternberg, 1978a;- 1978b);/ -Our- problem ^material As often ■ " 
considerably more complex than his Ihree- term syllogisms; but there are 
several places where the processes 'seem to be similar. Sternberg's 
"pivot, search," for example,' requires , the subject to establish a term 
which will permit' the combination 'of two premises into a* single ordered 
array; this may take several seconds, in his situation. Our solvers 
must find logical "pivots" too, but the number of admi|sable alterna- 
tives is often much larger than in the three- term case^ so discovery 
may require many seconds, or even some minutes. , In Sternberg's -'models, 
•the. "aval lability" of a solution element is sometimes a function of . 



memory from.the ImmediateTy preceding operation. This factor might be 
Operative In our "logic" problems. If sentences are phrased or "bunched" 
S<nS5»to facilitate 'appreciation of a dimension (age,, order, etc.)v 
then increased avaiTability of inclusions and exclusions should show 
up in the problem matrix. We have tried a few problem variations of 
this kind on srrtall grab samples of subjects, and occasional l>rih effect 
can be obtained. In one version of our "Murderer" problem (Bond, ' 
GabrieHi, &. Rigney, 1977), we put certain sentences with a,shared 
l^dimension next to each other, in the hope that appreciation of that 
Li mens ion would be enhanced. The effect we found was only slight; but 
tnl^re is so much complexity in our problems, and "so many ways for the 
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subject, to work, that such effects..are often masked -by the variability. 
'Sternberg- gets much, of his'data from highly practiced ^subjects on con- .-. 
. strained problems; -'so perhaps a "componential" attack' on our logic 
problems would p.roceed best with expert solvers.. - 

■ One of the most .powerful features of Sternberg's work is .his esti- 
mation of the accuracy of various. performance models; he gets R^'s oh 
the order of .90 and better for some of his models. Sternberg used the 
R parameters to decide which of several models -is best, and to suggest 
what proportion of variance is yet to be explained .-^ It .is possibli to 
predict performance in our logic problems rather well .from sentence-" 
logic extraction activity, as we have seen above. But it is also 
possible to predict general. 'difficul ty of the problems from superficial 
features, such as n^Hnber' of sentences in the problem, number of variables, 
and number of sentence combinations required for solution; we. get.rU in 
the 60's or higher, for this kind of prediction. 

' ' ■ -37- . ■ ' 



; We beTieve that a measure of logical . "depth" is; needed for -the : 
inferences in -our problems* and we are now exploring a simple three-' 
level -depth score. If this scoring scheme works,, then probability of " 
solution in a given problem' might be a function of a person.' s- a verage j- 
"realized depth" in several similar problems. .Given a person's ability 
in perceiving relations at various logical depths, we could model his/ ' 
her performance on any given -problem, via the depth parameters of the 
problem, and a. digital simulation pniqr^. . 

.^Data from our problems have some significance for individual differ 
ences in cognition. High-verbarability people ^tend to be faster at 
making, "name" marches than are low-verbal ability subjects (Hunt, Frost, 
&,Lunneborg,- .1973); ^and high-verbals are also much faster than, people' 
with low-verbal ability in jnaking taxonpmic category' matches and homo- . 
phone identity matches (Goldberg, Schwartz, & Stewart, 197.7). -Our re- 
sults suggest that appreciation of the strict logical relations implied 
in a sentence is. also related to verlDal ability. Perhaps the . "logic 
encoding rate" will turn out to be a useful information-prbcessing- • 
parameter for individuals. 

Hayes et al ( 1-977) employed logic problems in a. simulation study ' 
of human reasoning. Their work is in the Carnegie^Mellon "production - \ 
system" tradition. First,, they asked ^human subjects to -make relevancy 
judgments about the material in each of the problem sentences. As 
• expected^ the subjects were able to ignore extraneous ' information, and ; 
to foc.us on the key -elements of thejproblem. The investigators then • 
hypothesized that three problem-structuring processes were operating: • 
(1) a SETS heuristic which identifies groups. of items; (2) a TIME heur- . 
.istic which" tags, items containing- time-related, phrases' such as "yesterday; 



and. {3) a J3UESTI0N heuristic which places great relevancy-value on 
-items which are in a qiJery mode, anfd which tend: to define a solution . 
to- the" problem; A SNOBOL computer, program -Was written to imitateythese 
processes, and the program output of "relevancy" or "meaning" resembled 
the human judgments /rather 'closely. ." " 

Our data certainly confirm the critical impprta.nce of activities 
like sets; and the powerful orienting effects of QUESTION-lsehtences'. 
Indee'd, we believe that skill in" rapidly, defining sets, and in separ- 
ating' Targe. 5ets into subsets, is a distinctive mark of the good prob- 
letn soTver. For an inexperienced subject,' the set-defining activities, 
may be observed from the very Ijeginning of the solution ''process.. There 
is -also a close resemblance- between the Hayes JIME heuristic and our 
"ordering" variable.' One of the first things an expert solver looks 
'for is information about ordered arrays in the^, sentences. And if. order 
data are there, the expert generally hopes -to find that one or two prob- 
Tern, sentences are especially rich' in the order "subset" logic; often, 
too, the "bi^ ordlering- sentence leads rather directly to "smaller"' 
exclusion, statements; so- the sentence-combining' decisions are easier. ' 

Our seven trained subjects, we believe, would 'sol ve~ the Hayes • - 
"ATI -sports " problems very quickly. This :'is becausje fill -the sets and 
relations are obvious (even though some are irrelevant), and because, 

•our matrix skill's would afford rapid exclusions of the (few) name-sport 

- ■ • ■ ■ ' ' 

pairings. ' . ' 

* * ^ " ^ , ■ . *. . ■' 

• There appear^to be no.intr^insic barrier to coding problem sen- 
tences for their semantic contents,' w.ith a relatively full listing of 



meaning categories for each term 



in a sentence. If this can actually^ 



be done, then praduction-system models 'and aiding programs could G&p- 
■ ture. the essential behaviors in logical -problem' solving. The results, - 
of the present study indicate that the actual processing of problem . 
data can be routin.Tzed, bnce suitable inputs are recorded into- a matrix, 

or into some' other kind of memory. We know how -to teach, and to assist 

, •. . • . ■ 

via, computer, thii logical -inference part of "the problem. But there are 
stilT.many technica.1 issues in getting good dffta from the sentences 
into the logical processor; and these must be mastered before, an aid^ed 
probTem-solving system can be confident of solution in really different 
problems,. / 
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Five^. women received special honors at the annual awards luncheon of 
the ladies hospital auxiliary, . . ^ -^.X 

1\. Ann and^Mrs.Trask, eath work one day a week; Bea, Ms. Qui nn,; and 
the womai^ who received the ribbon all work two days; each day of 
the weekHs worked by at least one of .the five- 

2. The woman who received the ribbon works with Ms. Ross on Tuesdays 
and^with the armband winner.on Mondays. ' 

3.. The woman who received the certificate sees Eva at work "every 
. Friday and'Mrs. Sussman every Tuesday. . / * 
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A 



The five men in order pf their salaries, from least to 
greatest ^are: the Arm.y's employee, Mark jwhose las name is not 
Reiner), the shoe- salesman, David, and Mr/ Di-xon (who does not sell 
jewelry. ) - • ^ - . 
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Modified Schwartz Problem 



8. 
10. 

n, 

12. 
13. 



1. The hyena's owner- doesn' t live in the white, yellow, red, or green . house. 

2. Neither the Japanese, the Indian,, nor the Englishman lives in the green 
house. - • . 

3. Neither the American nor Canadian owns a zebra. . - .„ 

4. - The tea drinker doesn' t live in the blue house, and doesn't own a ' ' 
. turtle, hyena, ox, oi- horse. ' " 

5. Neither the Japanese nor the Englishman owns a hyena. 

6. The beer drinker isn't English, doesn't live in. the red house, and 
doesn't own an ox. ' v -.. ■ ■ ■ ■ ^ 

7. The zebra's owner doesn't live in the yellow or red house, and. doesn't 
drink milk. , ° ■' ' .. 

the coffee drin"ker doesn't own a zebra, or" an ox, live in the yellow, 
blue, or red house. - ~ 

The Japanese cioesn't live 'in the red, blue, gree, or yellow house- ' 
The American doesn' t-TiVe in^ the red,^blue, or green house! 
One of the men drinks whiskey. .. . - . ' ' ^ . 

The American does not drink, whiskey or milk and does not own a hjDpse. 
The rffi'lk drinker does not live in- a blue house- w " V 
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